{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "W3_Tutorial2.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "machine_shape": "hm",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/CIS-522/course-content/blob/main/tutorials/W03_MLPs/student/W3_Tutorial2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X2kPrtVyXtaP"
      },
      "source": [
        "# CIS-522 Week 3 Part 2\r\n",
        "# Multi-Layer Perceptrons (MLPs)\r\n",
        "\r\n",
        "__Instructor__: Konrad Kording\r\n",
        "\r\n",
        "__Content creators:__ Arash Ash"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p2DuJddRXxYY"
      },
      "source": [
        "---\r\n",
        "# Tutorial Objectives\r\n",
        "In the second tutorial of Week 3, we will dive deeper into MLPs and see more of their mathematical and practical aspects. Today we are going to see why MLPs are:\r\n",
        "\r\n",
        "* can evolve linearly in weights \r\n",
        "* the case of deep vs. wide\r\n",
        "* dependant on transfer functions\r\n",
        "* sensitive to initialization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e6GnDC12UtUD",
        "cellView": "form"
      },
      "source": [
        "#@markdown What is your Pennkey and pod? (text, not numbers, e.g. bfranklin)\n",
        "\n",
        "my_pennkey = 'value' #@param {type:\"string\"}\n",
        "my_pod = 'Select' #@param ['Select', 'euclidean-wombat', 'sublime-newt', 'buoyant-unicorn', 'lackadaisical-manatee','indelible-stingray','superfluous-lyrebird','discreet-reindeer','quizzical-goldfish','astute-jellyfish','ubiquitous-cheetah','nonchalant-crocodile','fashionable-lemur','spiffy-eagle','electric-emu','quotidian-lion']\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "NtskSZXZqu3W"
      },
      "source": [
        "# @title Slides\r\n",
        "from IPython.display import HTML\r\n",
        "HTML('<iframe src=\"https://docs.google.com/presentation/d/e/2PACX-1vSPvHqDTmMq4GyQy6lieNEFxq4qz1SmqC2RNoeei3_niECH53zneh8jJVYOnBIdk0Uaz7y2b9DK8V1t/embed?start=false&loop=false&delayms=3000\" frameborder=\"0\" width=\"480\" height=\"299\" allowfullscreen=\"true\" mozallowfullscreen=\"true\" webkitallowfullscreen=\"true\"></iframe>')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vgOQxVS1X2dB"
      },
      "source": [
        "---\r\n",
        "# Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xXd8Now_XwR7",
        "cellView": "form"
      },
      "source": [
        "# @title Imports\r\n",
        "import random\r\n",
        "import pathlib\r\n",
        "import random, time\r\n",
        "\r\n",
        "import torch\r\n",
        "import numpy as np\r\n",
        "import matplotlib.pyplot as plt\r\n",
        "\r\n",
        "import torch.nn as nn\r\n",
        "import torch.optim as optim\r\n",
        "import torch.nn.functional as F\r\n",
        "import torchvision.transforms as transforms\r\n",
        "from torchvision.datasets import ImageFolder\r\n",
        "from torch.utils.data import DataLoader, TensorDataset\r\n",
        "from torchvision.utils import make_grid\r\n",
        "from IPython.display import HTML, display\r\n",
        "\r\n",
        "dev = \"cpu\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZUyEQpxTSzxe",
        "cellView": "form"
      },
      "source": [
        "# @title Seeding for reproducibility\r\n",
        "seed = 522\r\n",
        "random.seed(seed)\r\n",
        "np.random.seed(seed)\r\n",
        "torch.manual_seed(seed)\r\n",
        "torch.cuda.manual_seed(seed)\r\n",
        "\r\n",
        "torch.backends.cudnn.deterministic = True\r\n",
        "torch.backends.cudnn.benchmark = False\r\n",
        "torch.set_deterministic(True)\r\n",
        "def seed_worker(worker_id):\r\n",
        "    worker_seed = seed % (worker_id+1)\r\n",
        "    np.random.seed(worker_seed)\r\n",
        "    random.seed(worker_seed)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "hj8C2Cf0G0i_"
      },
      "source": [
        "# @title Dataset download\r\n",
        "%%capture\r\n",
        "!rm -r AnimalFaces32x32/\r\n",
        "!git clone https://github.com/arashash/AnimalFaces32x32\r\n",
        "!rm -r afhq/\r\n",
        "!unzip ./AnimalFaces32x32/afhq_32x32.zip "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oRrTcZOlXA8C",
        "cellView": "form"
      },
      "source": [
        "# @title Figure Settings\r\n",
        "import ipywidgets as widgets\r\n",
        "%matplotlib inline \r\n",
        "fig_w, fig_h = (8, 6)\r\n",
        "plt.rcParams.update({'figure.figsize': (fig_w, fig_h)})\r\n",
        "%config InlineBackend.figure_format = 'retina'\r\n",
        "my_layout = widgets.Layout()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "40Qc7MFcX7Oc",
        "cellView": "form"
      },
      "source": [
        "# @title Helper functions\r\n",
        "def imshow(img):\r\n",
        "    img = img / 2 + 0.5     # unnormalize\r\n",
        "    npimg = img.numpy()\r\n",
        "    plt.imshow(np.transpose(npimg, (1, 2, 0)))\r\n",
        "    plt.axis(False)\r\n",
        "    plt.show()\r\n",
        "\r\n",
        "def color_grad(grad, M=500, x_max=1):\r\n",
        "  \"\"\"Plot gradient\r\n",
        "\r\n",
        "  \"\"\"\r\n",
        "  grad = grad.detach().cpu()\r\n",
        "  grad_colors = grad[:, 0]\r\n",
        "  grad_colors = (grad_colors / grad_colors.max() * 1e3).int() % 10\r\n",
        "  grad_colors = grad_colors.view(M, M).cpu().numpy()\r\n",
        "  return grad_colors\r\n",
        "\r\n",
        "def progress(epoch, loss, epochs=100):\r\n",
        "    return HTML(\"\"\"\r\n",
        "        <label for=\"file\">Training loss: {loss}</label>\r\n",
        "        <progress\r\n",
        "            value='{epoch}'\r\n",
        "            max='{epochs}',\r\n",
        "            style='width: 100%'\r\n",
        "        >\r\n",
        "            {epoch}\r\n",
        "        </progress>\r\n",
        "    \"\"\".format(loss=loss, epoch=epoch, epochs=epochs))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D1H4GG-0jWfy",
        "cellView": "form"
      },
      "source": [
        "# @title Part 1 Code\r\n",
        "class Net(nn.Module):\r\n",
        "    def __init__(self, actv, num_inputs, hidden_units, num_outputs):\r\n",
        "        super(Net, self).__init__()\r\n",
        "\r\n",
        "        exec('self.actv = nn.%s'%actv)\r\n",
        "\r\n",
        "        self.layers = nn.ModuleList()\r\n",
        "        for i in range(len(hidden_units)):\r\n",
        "          next_num_inputs = hidden_units[i] \r\n",
        "          self.layers += [nn.Linear(num_inputs, next_num_inputs)]\r\n",
        "          num_inputs = next_num_inputs\r\n",
        "\r\n",
        "        self.out = nn.Linear(num_inputs, num_outputs)\r\n",
        "\r\n",
        "    def forward(self, x):\r\n",
        "        # flattening\r\n",
        "        x = x.view(x.shape[0], -1)\r\n",
        "\r\n",
        "        for layer in self.layers:\r\n",
        "          x = self.actv(layer(x))\r\n",
        "        x = self.out(x)\r\n",
        "        return x\r\n",
        "\r\n",
        "K = 4\r\n",
        "sigma = 0.4\r\n",
        "N = 1000\r\n",
        "t = torch.linspace(0, 1, N)\r\n",
        "X = torch.zeros(K*N, 2)\r\n",
        "y = torch.zeros(K*N)\r\n",
        "for k in range(K):\r\n",
        "  X[k*N:(k+1)*N, 0] = t*(torch.sin(2*np.pi/K*(2*t+k)) + sigma**2*torch.randn(N))   # [TO-DO]\r\n",
        "  X[k*N:(k+1)*N, 1] = t*(torch.cos(2*np.pi/K*(2*t+k)) + sigma**2*torch.randn(N))   # [TO-DO]\r\n",
        "  y[k*N:(k+1)*N] = k\r\n",
        "\r\n",
        "# Shuffling\r\n",
        "shuffled_indeces = torch.randperm(K*N)\r\n",
        "X = X[shuffled_indeces]\r\n",
        "y = y[shuffled_indeces]\r\n",
        "\r\n",
        "# Test Train splitting\r\n",
        "test_size = int(0.2*N)\r\n",
        "X_test = X[:test_size]\r\n",
        "y_test = y[:test_size]\r\n",
        "X_train = X[test_size:]\r\n",
        "y_train = y[test_size:]\r\n",
        "\r\n",
        "\r\n",
        "batch_size = 128\r\n",
        "test_data = TensorDataset(X_test, y_test)\r\n",
        "test_loader = DataLoader(test_data, batch_size=batch_size,\r\n",
        "                         shuffle=False, num_workers=0)\r\n",
        "\r\n",
        "train_data = TensorDataset(X_train, y_train)\r\n",
        "train_loader = DataLoader(train_data, batch_size=batch_size, drop_last=True,\r\n",
        "                        shuffle=True, num_workers=0, worker_init_fn=seed_worker)\r\n",
        "\r\n",
        "def train_test_classification(net, criterion, optimizer,\r\n",
        "                              train_loader, test_loader,\r\n",
        "                              num_epochs=1, verbose=True, \r\n",
        "                              training_plot=False):\r\n",
        "  if verbose:\r\n",
        "    progress_bar = display(progress(0, 0, num_epochs), display_id=True)\r\n",
        "\r\n",
        "  net.train()\r\n",
        "  training_losses = []\r\n",
        "  for epoch in range(num_epochs):  # loop over the dataset multiple times\r\n",
        "      running_loss = 0.0\r\n",
        "      for i, data in enumerate(train_loader, 0):\r\n",
        "          # get the inputs; data is a list of [inputs, labels]\r\n",
        "          inputs, labels = data\r\n",
        "          inputs = inputs.to(dev).float()\r\n",
        "          labels = labels.to(dev).long()\r\n",
        "\r\n",
        "          # zero the parameter gradients\r\n",
        "          optimizer.zero_grad()\r\n",
        "\r\n",
        "          # forward + backward + optimize\r\n",
        "          outputs = net(inputs)\r\n",
        "\r\n",
        "          loss = criterion(outputs, labels)\r\n",
        "          loss.backward()\r\n",
        "          optimizer.step()\r\n",
        "\r\n",
        "          # print statistics\r\n",
        "          if verbose:\r\n",
        "            training_losses += [loss.item()]\r\n",
        "            running_loss += loss.item()\r\n",
        "            if i % 10 == 9:    # update every 10 mini-batches\r\n",
        "                progress_bar.update(progress(epoch+1, running_loss / 10, num_epochs))\r\n",
        "                running_loss = 0.0\r\n",
        "\r\n",
        "  net.eval()\r\n",
        "  def test(data_loader):\r\n",
        "    correct = 0\r\n",
        "    total = 0\r\n",
        "    for data in data_loader:\r\n",
        "        inputs, labels = data\r\n",
        "        inputs = inputs.to(dev).float()\r\n",
        "        labels = labels.to(dev).long()\r\n",
        "\r\n",
        "        outputs = net(inputs)\r\n",
        "        _, predicted = torch.max(outputs, 1)\r\n",
        "        total += labels.size(0)\r\n",
        "        correct += (predicted == labels).sum().item()\r\n",
        "\r\n",
        "    acc = 100 * correct / total\r\n",
        "    return total, acc\r\n",
        "\r\n",
        "  train_total, train_acc = test(train_loader)\r\n",
        "  test_total, test_acc = test(test_loader)\r\n",
        "\r\n",
        "  if verbose:\r\n",
        "    print('Accuracy on the %d training samples: %0.2f %%' % (train_total, train_acc))\r\n",
        "    print('Accuracy on the %d testing samples: %0.2f %%' % (test_total, test_acc))\r\n",
        "\r\n",
        "  if training_plot:\r\n",
        "    plt.plot(training_losses)\r\n",
        "    plt.xlabel('Batch')\r\n",
        "    plt.ylabel('Training loss')\r\n",
        "    plt.show()\r\n",
        "  \r\n",
        "  return train_acc, test_acc\r\n",
        "\r\n",
        "def sample_grid(M=500, x_max = 2.0):\r\n",
        "  ii, jj = torch.meshgrid(torch.linspace(-x_max, x_max,M),\r\n",
        "                          torch.linspace(-x_max, x_max, M))\r\n",
        "  X_all = torch.cat([ii.unsqueeze(-1),\r\n",
        "                     jj.unsqueeze(-1)],\r\n",
        "                     dim=-1).view(-1, 2)\r\n",
        "  return X_all\r\n",
        "\r\n",
        "def plot_decision_map(X_all, y_pred, X_test, y_test, M=500, x_max = 2.0, eps = 1e-3):\r\n",
        "  decision_map = torch.argmax(y_pred, dim=1)\r\n",
        "\r\n",
        "  for i in range(len(X_test)):\r\n",
        "    indeces = (X_all[:, 0] - X_test[i, 0])**2 + (X_all[:, 1] - X_test[i, 1])**2 < eps    # [TO-DO]\r\n",
        "    decision_map[indeces] = (K + y_test[i]).long()\r\n",
        "\r\n",
        "  decision_map = decision_map.view(M, M).cpu()\r\n",
        "  plt.imshow(decision_map, extent=[-x_max, x_max, -x_max, x_max], cmap='jet')\r\n",
        "  plt.plot()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SG3iPDUaxSLv"
      },
      "source": [
        "---\r\n",
        "# Section 1: Gradient visualization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CwszyupvkL3-",
        "cellView": "form"
      },
      "source": [
        "#@title Video : How ReLU Networks Work: Locally Linear Functions\n",
        "try: t1;\n",
        "except NameError: t1=time.time()\n",
        "\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"L6AmjfozoBc\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yRsWs55XfyAS"
      },
      "source": [
        "## Exercise 1: Implement gradient visualization\r\n",
        "The goal here is to visualize the gradient of the input with respect to the output. We can achieve this using the `backward()` function in PyTorch. Since we can only call `backward()` on a scaler, we could average the output tensor  to acheive this. And finally give the gradients of the input to the `plot_polytopes` function.\r\n",
        "\r\n",
        "**Note**: We have imported the necessary functions from the previous tutorial for you. So if you don't remember how they work, you could hover over the function or class and see the implementations."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ki3h4MiggLWJ"
      },
      "source": [
        "def run_grad_viz():\r\n",
        "  ####################################################################\r\n",
        "  # Fill in all missing code below (...),\r\n",
        "  # then remove or comment the line below to test your function\r\n",
        "  raise NotImplementedError(\"Define the grad visualization function\")\r\n",
        "  ####################################################################\r\n",
        "\r\n",
        "  # define a network which is randomly initialized, any ReLU model would do the trick!\r\n",
        "  rand_net = Net('ReLU()', X_train.shape[1], [128], K).to(dev)\r\n",
        "\r\n",
        "  # sample the grid to generate an input batch, hover over the tensor to see it's shape\r\n",
        "  X_all = sample_grid(x_max=1)\r\n",
        "\r\n",
        "  # we need to allow PyTorch to keep track of the input gradients\r\n",
        "  X_all.requires_grad = ...\r\n",
        "\r\n",
        "  # generate the output\r\n",
        "  outputs = rand_net(X_all)\r\n",
        "\r\n",
        "  # create a scaler value for backward()\r\n",
        "  loss = ...\r\n",
        "\r\n",
        "  # do the backgrop\r\n",
        "  loss.backward()\r\n",
        "\r\n",
        "  # defines a color where gradient is constant\r\n",
        "  grad_colors = color_grad(...)\r\n",
        "\r\n",
        "  return grad_colors\r\n",
        "\r\n",
        "# Uncomment below to test your function\r\n",
        "# grad_colors = run_grad_viz()\r\n",
        "# plt.imshow(grad_colors, cmap='rainbow')\r\n",
        "# plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "16ty9XK-mz5T"
      },
      "source": [
        "[*Click for solution*](https://github.com/CIS-522/course-content/blob/main/tutorials/W03_MLPs/solutions/W3_Tutorial2_Solution_Ex01.py)\n",
        "\n",
        "*Example output:*  \n",
        "\n",
        "\n",
        "<img alt='Solution hint 1' align='left' width=416 height=416 src=https://raw.githubusercontent.com/CIS-522/course-content/main/tutorials/W03_MLPs/static/W3_Tutorial2_Solution_Ex01.png />\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "303vZSEakhuq"
      },
      "source": [
        "#@markdown What would happen if the loss function were non-linear (e.g., cross-entropy)? Would we still see polytopes?\r\n",
        "w3_with_non_linear_loss = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QZS17RNAkbmt"
      },
      "source": [
        "## Implement gradient visualization that flows! (optional)\r\n",
        "\r\n",
        "we saw that gradients are constant if the network is linear, let's visualize the gradient's dimension now. The gradient has the same dimension as its tensor. Therefore, the gradient of the input has the same dimension as the input, which in this case is 2. So we can visualize it using 2D vectors using `plt.quiver` function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t8R71qCfkWhM"
      },
      "source": [
        "def plot_grad(X_all, grad, y_test, M=500):\r\n",
        "  grad = grad.detach().cpu()\r\n",
        "  X_all = X_all.detach().cpu()\r\n",
        "  y_test = y_test.detach().cpu()\r\n",
        "\r\n",
        "  plt.quiver(X_all[:, 0], X_all[:, 1],\r\n",
        "             grad[:, 0], grad[:, 1], y_test)\r\n",
        "  plt.show()\r\n",
        "\r\n",
        "rand_net = Net('ReLU()', X_train.shape[1], [128], K).to(dev)\r\n",
        "X_test = X_test.to(dev)\r\n",
        "y_test = y_test.to(dev)\r\n",
        "X_test.requires_grad = True\r\n",
        "\r\n",
        "outputs = rand_net(X_test)\r\n",
        "\r\n",
        "loss = torch.mean(outputs)\r\n",
        "loss.backward()\r\n",
        "\r\n",
        "plot_grad(X_test, X_test.grad, y_test)\r\n",
        "\r\n",
        "# we need to reset it to not interfere later\r\n",
        "X_test.requires_grad = False "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lkuxYZ_-lJIr"
      },
      "source": [
        "---\n",
        "# Section 2: MLPs vs Polynomial regression"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bn7efLdVmz8Z",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Polynomial Features\n",
        "\n",
        "try: t2;\n",
        "except NameError: t2=time.time()\n",
        "\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"90gSC6Vwg1I\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7VJgKVfGlaW_"
      },
      "source": [
        "## Exercise 2: Add polynomial features and train without any hidden layers\r\n",
        "Let's use the same Spiral dataset generated before with two features. And then add more polynomial features. And finally, train a single Linear layer. We could use the same MLP network with no hidden layers (though it would not be called an MLP anymore!).\r\n",
        "\r\n",
        "Note that we will add polynomial terms upto $P=50$ which means that for every $x_1^n x_2^m$ term, $n+m\\leq P$. Now it's a good math excercise to prove why the total number of polynomial features upto $P$ becomes,\r\n",
        "$$\r\n",
        "\\text{# of terms} = \\frac{(P+1)(P+2)}{2}\r\n",
        "$$\r\n",
        "\r\n",
        "Also, we don't need the polynomial term with degree zero (which is the constatnt term) since `nn.Linear` layers have bias terms. Therefore we will have one fewer polynomial feature."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OEn2QUQkzzzv"
      },
      "source": [
        "def run_poly_clasification(poly_degree):\r\n",
        "  ####################################################################\r\n",
        "  # Fill in all missing code below (...),\r\n",
        "  # then remove or comment the line below to test your function\r\n",
        "  raise NotImplementedError(\"Define the poly clasification function\")\r\n",
        "  ###################################################################\r\n",
        "\r\n",
        "  def make_poly_features(poly_degree, X):\r\n",
        "    # Define the number of polynomial features except the bias term\r\n",
        "    num_features = ...\r\n",
        "    poly_X = torch.zeros((X.shape[0], num_features))\r\n",
        "    count = 0\r\n",
        "    for i in range(poly_degree+1):\r\n",
        "      for j in range(poly_degree+1):\r\n",
        "         # no need to add zero degree since model has biases\r\n",
        "        if j+i > 0:\r\n",
        "          if j+i <= poly_degree:\r\n",
        "            # Define the polynomial term\r\n",
        "            poly_X[:, count] = ...\r\n",
        "            count += 1\r\n",
        "    return poly_X, num_features\r\n",
        "\r\n",
        "  poly_X_test, num_features = make_poly_features(poly_degree, X_test)\r\n",
        "  poly_X_train, _ = make_poly_features(poly_degree, X_train)\r\n",
        "\r\n",
        "  batch_size = 128\r\n",
        "  poly_test_data = TensorDataset(poly_X_test, y_test)\r\n",
        "  poly_test_loader = DataLoader(poly_test_data, batch_size=batch_size,\r\n",
        "                          shuffle=False, num_workers=1)\r\n",
        "  poly_train_data = TensorDataset(poly_X_train, y_train)\r\n",
        "  poly_train_loader = DataLoader(poly_train_data, batch_size=batch_size,\r\n",
        "                          shuffle=True, num_workers=1)\r\n",
        "  \r\n",
        "  # define a linear model using MLP class\r\n",
        "  poly_net = ...\r\n",
        "\r\n",
        "  # Train it!\r\n",
        "  criterion = nn.CrossEntropyLoss()\r\n",
        "  optimizer = optim.Adam(poly_net.parameters(), lr=1e-3)\r\n",
        "  _, _ = train_test_classification(poly_net, criterion, optimizer, \r\n",
        "                                  poly_train_loader, poly_test_loader,\r\n",
        "                                  num_epochs=100)\r\n",
        "  # Test it\r\n",
        "  X_all = sample_grid()\r\n",
        "  poly_X_all, _ = make_poly_features(poly_degree, X_all)\r\n",
        "  y_pred = poly_net(poly_X_all)\r\n",
        "\r\n",
        "  # Plot it\r\n",
        "  plot_decision_map(X_all, y_pred, X_test, y_test)\r\n",
        "  plt.show()\r\n",
        "\r\n",
        "  return num_features\r\n",
        "\r\n",
        "# Uncomment below to test your function\r\n",
        "# max_poly_degree = 50\r\n",
        "# num_features = run_poly_clasification(max_poly_degree)\r\n",
        "# print('Number of features: %d'%num_features)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wIRgagiantFn"
      },
      "source": [
        "[*Click for solution*](https://github.com/CIS-522/course-content/blob/main/tutorials/W03_MLPs/solutions/W3_Tutorial2_Solution_Ex02.py)\n",
        "\n",
        "*Example output:*  \n",
        "\n",
        "```\n",
        "Accuracy on the 3800 training samples: 72.87 %\n",
        "Accuracy on the 200 testing samples: 74.50 %\n",
        "Number of features: 1325\n",
        "```\n",
        "\n",
        "<img alt='Solution hint 2' align='left' width=416 height=416 src=https://raw.githubusercontent.com/CIS-522/course-content/main/tutorials/W03_MLPs/static/W3_Tutorial2_Solution_Ex02.png />\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "uEjJla5K0vfm"
      },
      "source": [
        "#@markdown Do you think this model is performing well outside its training distribution? Why?\r\n",
        "w3_poly_OoD = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CJYcwHzugx6N"
      },
      "source": [
        "---\n",
        "# Section 3: Wider vs deeper networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5cXchBdhg_r7",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Expressivity vs Learnability\n",
        "\n",
        "try: t3;\n",
        "except NameError: t3=time.time()\n",
        "\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"H18gs1z2PTw\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FG1bGJhXaIpQ"
      },
      "source": [
        "## Exercise 3: Wide vs. Deep while keeping number of parameters same\r\n",
        "Let's find the optimal number of hidden layers under a fixed number of parameters constraint!\r\n",
        "\r\n",
        "But first, we need a model parameter counter. You could iterate over the model layers by calling `.parameters()` and then use `.numel()` to count the layer parameters. Also, you can use `requires_grad` attribute to make sure it's a trainable parameter.\r\n",
        "\r\n",
        "After defining the counter function, we will step by step increase the depth and then iterate over the possible number of hidden units (assuming same for all hidden layers); then using our parameter counter choose the number of hidden units that results in overall close to `max_par_count` parameters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XXTmr99LAA21"
      },
      "source": [
        "def run_depth_optimizer(max_par_count,  max_hidden_layer):\r\n",
        "  ####################################################################\r\n",
        "  # Fill in all missing code below (...),\r\n",
        "  # then remove or comment the line below to test your function\r\n",
        "  raise NotImplementedError(\"Define the depth optimizer function\")\r\n",
        "  ###################################################################\r\n",
        "  def count_parameters(model):\r\n",
        "    par_count = 0\r\n",
        "    for p in model.parameters():\r\n",
        "      if p.requires_grad:\r\n",
        "        par_count = ...\r\n",
        "    return par_count\r\n",
        "    \r\n",
        "  # define the list of number of hidden layers to try\r\n",
        "  hidden_layers = ...\r\n",
        "\r\n",
        "  # test test score list\r\n",
        "  test_scores = []\r\n",
        "\r\n",
        "  for hidden_layer in hidden_layers:\r\n",
        "    # Initialize the hidden units in each hidden layer to be 1\r\n",
        "    hidden_units = np.ones(hidden_layer, dtype=np.int)\r\n",
        "\r\n",
        "    # Define the the with hidden units equal to 1\r\n",
        "    wide_net = Net('ReLU()', X_train.shape[1], hidden_units, K).to(dev)\r\n",
        "    par_count = count_parameters(wide_net)\r\n",
        "\r\n",
        "    # icreament hidden_units and repeat until the par_count reaches the desired count\r\n",
        "    while par_count < max_par_count: \r\n",
        "      hidden_units += 1\r\n",
        "      wide_net = Net('ReLU()', X_train.shape[1], hidden_units, K).to(dev)\r\n",
        "      par_count = ...\r\n",
        "\r\n",
        "    # Train it\r\n",
        "    criterion = nn.CrossEntropyLoss()\r\n",
        "    optimizer = optim.Adam(wide_net.parameters(), lr=1e-3)\r\n",
        "    _, test_acc = train_test_classification(wide_net, criterion, optimizer, train_loader,\r\n",
        "                                            test_loader, num_epochs=100)\r\n",
        "    test_scores += [test_acc]\r\n",
        "\r\n",
        "  return hidden_layers, test_scores\r\n",
        "\r\n",
        "# Uncomment below to test your function\r\n",
        "# max_par_count = 100\r\n",
        "# max_hidden_layer = 5\r\n",
        "# hidden_layers, test_scores = run_depth_optimizer(max_par_count,  max_hidden_layer)\r\n",
        "\r\n",
        "# plt.xlabel('# of hidden layers')\r\n",
        "# plt.ylabel('Test accuracy')\r\n",
        "# plt.plot(hidden_layers, test_scores)\r\n",
        "# plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hgWsXwZYoYs1"
      },
      "source": [
        "[*Click for solution*](https://github.com/CIS-522/course-content/blob/main/tutorials/W03_MLPs/solutions/W3_Tutorial2_Solution_Ex03.py)\n",
        "\n",
        "*Example output:*  \n",
        "\n",
        "```\n",
        "Training loss: 0.37049919962882993\n",
        "Accuracy on the 3712 training samples: 84.89 %\n",
        "Accuracy on the 200 testing samples: 87.50 %\n",
        "---\n",
        "Training loss: 0.3012431740760803 \n",
        "Accuracy on the 3712 training samples: 92.13 %\n",
        "Accuracy on the 200 testing samples: 91.50 %\n",
        "---\n",
        "Training loss: 0.22815748304128647\n",
        "Accuracy on the 3712 training samples: 94.18 %\n",
        "Accuracy on the 200 testing samples: 94.50 %\n",
        "---\n",
        "Training loss: 0.47335213124752046\n",
        "Accuracy on the 3712 training samples: 83.11 %\n",
        "Accuracy on the 200 testing samples: 85.00 %\n",
        "```\n",
        "\n",
        "<img alt='Solution hint 3' align='left' width=416 height=416 src=https://raw.githubusercontent.com/CIS-522/course-content/main/tutorials/W3_MLPs/static/W3_Tutorial2_Solution_Ex03.png />\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "w1X-jo7JEalh"
      },
      "source": [
        "#@markdown Here we see that three hidden layers seem to be optimum. Why do you think increasing hidden layers after a certain point hurt in this scenario?\r\n",
        "w3_why_three = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vfY1h-lFndjt"
      },
      "source": [
        "---\n",
        "# Section 4: Linear Learning in Wide Multi-Layer Perceptrons"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MNj8KuO7nqIP",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Neural Tangent Kernels\n",
        "\n",
        "try: t4;\n",
        "except NameError: t4=time.time()\n",
        "\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"5Lj6kp4k7Sk\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oPKms1XJWboJ"
      },
      "source": [
        "## Exercise 4: Motivation for NTKs\r\n",
        "lazy training of overcomplete MLPs results in linear changes in weights. Let's try to see it here.\r\n",
        "\r\n",
        "We will train a wide MLP step by step with small learning rates (when the learning rate is low, it's called lazy or linear training!) and keep track of some of the weight to see how they change.\r\n",
        "\r\n",
        "Note that you could index the network layers like a list (since we defined it with ModuleList). You could access the weights and biases of a `nn.Linear` layer by getting `.weight` and `.bias` attributes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fx1PQGlOOeDh"
      },
      "source": [
        "def run_lazy_training(num_time_steps, num_select_weights, step_epoch):\r\n",
        "  ####################################################################\r\n",
        "  # Fill in all missing code below (...),\r\n",
        "  # then remove or comment the line below to test your function\r\n",
        "  raise NotImplementedError(\"Define the lazy training function\")\r\n",
        "  ###################################################################\r\n",
        "\r\n",
        "  # Define a wide MLP\r\n",
        "  net = ...\r\n",
        "  criterion = nn.CrossEntropyLoss()\r\n",
        "  optimizer = optim.SGD(net.parameters(), lr=1e-2)\r\n",
        "\r\n",
        "  # let's save only couple of parameters at each time step\r\n",
        "  weights = torch.zeros(num_time_steps, num_select_weights)\r\n",
        "  for i in range(num_time_steps):\r\n",
        "    _, _ = train_test_classification(net, criterion, optimizer, train_loader,\r\n",
        "                                    test_loader, num_epochs=step_epoch, verbose=False)\r\n",
        "    \r\n",
        "    # let's pratice some tensor navigations!\r\n",
        "    # access the first layer weights\r\n",
        "    # and index the first column \r\n",
        "    # and slice upto num_select_weights paramaeters\r\n",
        "    weights[i] = ...\r\n",
        "\r\n",
        "  return weights\r\n",
        "\r\n",
        "# Uncomment below to test your function\r\n",
        "# num_select_weights = 10\r\n",
        "# num_time_steps = 5\r\n",
        "# step_epoch = 50\r\n",
        "# weights = run_lazy_training(num_time_steps, num_select_weights, step_epoch)\r\n",
        "# for k in range(num_select_weights):\r\n",
        "#   weight = weights[:, k].detach()\r\n",
        "#   epochs = range(1, 1+num_time_steps*step_epoch, step_epoch)\r\n",
        "#   plt.plot(epochs, weight, label='weight #%d'%k)\r\n",
        "    \r\n",
        "# plt.xlabel('epochs')\r\n",
        "# plt.legend()\r\n",
        "# plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WQaiIacfpCrv"
      },
      "source": [
        "[*Click for solution*](https://github.com/CIS-522/course-content/blob/main/tutorials/W03_MLPs/solutions/W3_Tutorial2_Solution_Ex04.py)\n",
        "\n",
        "*Example output:*  \n",
        "\n",
        "\n",
        "<img alt='Solution hint 4' align='left' width=416 height=416 src=https://raw.githubusercontent.com/CIS-522/course-content/main/tutorials/W03_MLPs/static/W3_Tutorial2_Solution_Ex04.png />\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VYMYH7l7RdIm"
      },
      "source": [
        "## Neural Tangent Kernels (NTKs)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "HuPeo0rNQlzK"
      },
      "source": [
        "#@title Video: Neural Tangent Kernels\r\n",
        "\r\n",
        "from IPython.display import YouTubeVideo\r\n",
        "video = YouTubeVideo(id=\"rjbpZCSuMgQ\", width=854, height=480, fs=1)\r\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\r\n",
        "\r\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "-vO_oz4XPH4d"
      },
      "source": [
        "#@markdown How could we speed up lazy training?\r\n",
        "w3_fast_NTK = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J8lbbH6upXg6"
      },
      "source": [
        "---\n",
        "# Section 5: Deeper MLPs"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JIwa78eMpxwL",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Data Augmentation\n",
        "\n",
        "try: t5;\n",
        "except NameError: t5=time.time()\n",
        "\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"RtcJ8gn2kj0\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mk_MnhzRDw2t"
      },
      "source": [
        "## Exercise 5: Dataloader on a real-world dataset\r\n",
        "Let's build our first real-world dataset loader with Data Preprocessing and Augmentation! And we will use the Torchvision transforms to do it.\r\n",
        "\r\n",
        "We'd like to have a simple data augmentation with the following steps:\r\n",
        "* Random rotation with 10 degrees (RandomRotation)\r\n",
        "* Random horizontal flipping (RandomHorizontalFlip)\r\n",
        "\r\n",
        "and we'd like a preprocessing that:\r\n",
        "* makes Pytorch tensors in the range [0, 1] (ToTensor)\r\n",
        "* normalizes the input in the range [-1, 1] (Normalize)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "agJn1BtHIB1x"
      },
      "source": [
        "def get_data_loaders(batch_size):\r\n",
        "  ####################################################################\r\n",
        "  # Fill in all missing code below (...),\r\n",
        "  # then remove or comment the line below to test your function\r\n",
        "  raise NotImplementedError(\"Define the get data loaders function\")\r\n",
        "  ###################################################################\r\n",
        "\r\n",
        "  # define the list of transforms done only during training\r\n",
        "  augmentation_transforms = ...\r\n",
        "\r\n",
        "  # define the list of transforms done in training and testing (after augmentation)\r\n",
        "  preprocessing_transforms = ...\r\n",
        "  \r\n",
        "  # compose them together\r\n",
        "  train_transform = transforms.Compose(augmentation_transforms + preprocessing_transforms)\r\n",
        "  test_transform = transforms.Compose(preprocessing_transforms)\r\n",
        "\r\n",
        "  # using pathlib to be compatible with all OS's\r\n",
        "  data_path = pathlib.Path('.')/'afhq'\r\n",
        "\r\n",
        "  # define the dataset objects (they can load one by one)\r\n",
        "  img_train_dataset = ImageFolder(data_path/'train', transform=train_transform)\r\n",
        "  img_test_dataset = ImageFolder(data_path/'val', transform=test_transform)\r\n",
        "\r\n",
        "  # define the dataloader objects (they can load batch by batch)\r\n",
        "  img_train_loader = DataLoader(img_train_dataset, batch_size=batch_size,\r\n",
        "                                shuffle=True, num_workers=1, worker_init_fn=seed_worker)\r\n",
        "  # num_workers can be set to higher if running on Colab Pro TPUs to speed up,\r\n",
        "  # with more than one worker, it will do multithreading to queue batches\r\n",
        "  img_test_loader = DataLoader(img_test_dataset, batch_size=batch_size,\r\n",
        "                          shuffle=False, num_workers=1)\r\n",
        "  \r\n",
        "  return img_train_loader, img_test_loader\r\n",
        "  \r\n",
        "# Uncomment below to test your function\r\n",
        "# batch_size = 64\r\n",
        "# img_train_loader, img_test_loader = get_data_loaders(batch_size)\r\n",
        "# # get some random training images\r\n",
        "# dataiter = iter(img_train_loader)\r\n",
        "# images, labels = dataiter.next()\r\n",
        "# # show images\r\n",
        "# imshow(make_grid(images, nrow=8))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BoUs2WFJpkxC"
      },
      "source": [
        "[*Click for solution*](https://github.com/CIS-522/course-content/blob/main/tutorials/W03_MLPs/solutions/W3_Tutorial2_Solution_Ex05.py)\n",
        "\n",
        "*Example output:*  \n",
        "\n",
        "<img alt='Solution hint 5' align='left' width=416 height=416 src=https://raw.githubusercontent.com/CIS-522/course-content/main/tutorials/W03_MLPs/static/W3_Tutorial2_Solution_Ex05.png />\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "NT8lU-wnBcIT"
      },
      "source": [
        "#@title Video: Classifying Animal Functions\r\n",
        "\r\n",
        "from IPython.display import YouTubeVideo\r\n",
        "video = YouTubeVideo(id=\"JHJQZCD0mhA\", width=854, height=480, fs=1)\r\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\r\n",
        "\r\n",
        "video\r\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "keBtoq559ceG"
      },
      "source": [
        "# Train it\r\n",
        "net = Net('ReLU()', 3*32*32, [64, 64, 64], 3).to(dev) \r\n",
        "criterion = nn.MultiMarginLoss(margin=1.0)\r\n",
        "optimizer = optim.Adam(net.parameters(), lr=3e-4)\r\n",
        "_, _ = train_test_classification(net, criterion, optimizer,\r\n",
        "                                img_train_loader, img_test_loader,\r\n",
        "                                num_epochs=30)\r\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "xLiQNxB_CVrt"
      },
      "source": [
        "#@title Video: Map Receptive Fields\r\n",
        "\r\n",
        "from IPython.display import YouTubeVideo\r\n",
        "video = YouTubeVideo(id=\"YcnImVQtaqc\", width=854, height=480, fs=1)\r\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\r\n",
        "\r\n",
        "video\r\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "371o0g0ZCOIu"
      },
      "source": [
        "# visualize the feature map\r\n",
        "fc1_weights = net.layers[0].weight.view(64, 3, 32, 32).detach().cpu()\r\n",
        "fc1_weights /= torch.max(torch.abs(fc1_weights))\r\n",
        "imshow(make_grid(fc1_weights, nrow=8))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "_ePuGQjC65FM"
      },
      "source": [
        "#@markdown Even though it's three layers deep, we see distinct animal faces in the first layer feature map. Do you think this MLP has a hierarchical feature representation? why?\r\n",
        "w3_isHierarchical = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_aQSP-_Yy4Hm"
      },
      "source": [
        "---\n",
        "# Section 6: The choice of transfer function matters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zFG-Oz1P3FoS",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Transfer Functions\n",
        "try: t6;\n",
        "except NameError: t6=time.time()\n",
        "\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"UTQuxWSfuUM\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fc6M5Ubf4mIo"
      },
      "source": [
        "## Exercise 6: Find the best transfer function for this model\r\n",
        "Refer [here](https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity) for their documentations."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DYmgQtA534dn"
      },
      "source": [
        "# Possible Activations\r\n",
        "Activations = ['ReLU', 'Tanh', 'Sigmoid', 'ELU', 'Hardshrink', 'Hardsigmoid',\r\n",
        "'Hardtanh', 'Hardswish', 'LeakyReLU', 'LogSigmoid', 'PReLU',\r\n",
        "'ReLU6', 'RReLU', 'SELU', 'CELU', 'GELU', 'SiLU', 'Softplus',\r\n",
        "'Softshrink', 'Softsign', 'Tanhshrink']\r\n",
        "\r\n",
        "your_picks = ['ReLU'] # <--- you can try multiple picks and report the best\r\n",
        "\r\n",
        "for actv in your_picks:\r\n",
        "  print(actv)\r\n",
        "  actv = actv+'()'\r\n",
        "  net = Net(actv, 3*32*32, [128, 32], 3).to(dev) \r\n",
        "  criterion = nn.MultiMarginLoss(margin=1.0)\r\n",
        "  optimizer = optim.Adam(net.parameters(), lr=3e-4)\r\n",
        "  _, _ = train_test_classification(net, criterion, optimizer,\r\n",
        "                                  img_train_loader, img_test_loader,\r\n",
        "                                  num_epochs=20)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "1wHNXw2IEYiF"
      },
      "source": [
        "#@markdown Which activation function did you choose? Why?\r\n",
        "w3_whichActiv = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QaE5OQ_39jRH"
      },
      "source": [
        "---\n",
        "# Section 7: The need for good initialization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fsZzbd099ltE",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Initialization\n",
        "try: t7;\n",
        "except NameError: t7=time.time()\n",
        "\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"C7NdUgg40YY\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "40KCBDx686kr"
      },
      "source": [
        "## Xavier initialization\r\n",
        "Let us look at the scale distribution of an output (e.g., a hidden variable)  $o_i$  for some fully-connected layer without nonlinearities. With  $n_{in}$  inputs  ($x_j$)  and their associated weights  $w_{ij}$  for this layer. Then an output is given by,\r\n",
        "$$\r\n",
        "o_{i} = \\sum_{j=1}^{n_\\mathrm{in}} w_{ij} x_j\r\n",
        "$$\r\n",
        "The weights  $w_{ij}$  are all drawn independently from the same distribution. Furthermore, let us assume that this distribution has zero mean and variance  $\\sigma^2$ . Note that this does not mean that the distribution has to be Gaussian, just that the mean and variance need to exist. For now, let us assume that the inputs to the layer  $x_j$ also have zero mean and variance  $\\gamma^2$  and that they are independent of $w_{ij}$ and independent of each other. In this case, we can compute the mean and variance of $o_i$ as follows:\r\n",
        "\\begin{split}\\begin{aligned}\r\n",
        "    E[o_i] & = \\sum_{j=1}^{n_\\mathrm{in}} E[w_{ij} x_j] = \\sum_{j=1}^{n_\\mathrm{in}} E[w_{ij}] E[x_j] = 0, \\\\\r\n",
        "    \\mathrm{Var}[o_i] & = E[o_i^2] - (E[o_i])^2 = \\sum_{j=1}^{n_\\mathrm{in}} E[w^2_{ij} x^2_j] - 0 = \\sum_{j=1}^{n_\\mathrm{in}} E[w^2_{ij}] E[x^2_j] = n_\\mathrm{in} \\sigma^2 \\gamma^2\r\n",
        "\\end{aligned}\\end{split}\r\n",
        "One way to keep the variance fixed is to set $n_{in}\\sigma^2=1$ . Now consider backpropagation. There we face a similar problem, albeit with gradients being propagated from the layers closer to the output. Using the same reasoning as for forward propagation, we see that the gradients’ variance can blow up unless $n_{out}\\sigma^2=1$ , where  $n_{out}$ is the number of outputs of this layer. This leaves us in a dilemma: we cannot possibly satisfy both conditions simultaneously. Instead, we simply try to satisfy:\r\n",
        "\\begin{aligned}\r\n",
        "\\frac{1}{2} (n_\\mathrm{in} + n_\\mathrm{out}) \\sigma^2 = 1 \\text{ or equivalently }\r\n",
        "\\sigma = \\sqrt{\\frac{2}{n_\\mathrm{in} + n_\\mathrm{out}}}\r\n",
        "\\end{aligned}\r\n",
        "This is the reasoning underlying the now-standard and practically beneficial Xavier initialization, named after the first author of its creators [Glorot & Bengio, 2010]. Typically, the Xavier initialization samples weights from a Gaussian distribution with zero mean and variance  $\\sigma^2=\\frac{2}{(n_{in}+n_{out})}$. We can also adapt Xavier’s intuition to choose the variance when sampling weights from a uniform distribution. Note that the uniform distribution $U(−a,a)$ has variance $\\frac{a^2}{3}$. Plugging this into our condition on $\\sigma^2$ yields the suggestion to initialize according to\r\n",
        "$$\r\n",
        "U\\left(-\\sqrt{\\frac{6}{n_\\mathrm{in} + n_\\mathrm{out}}}, \\sqrt{\\frac{6}{n_\\mathrm{in} + n_\\mathrm{out}}}\\right)\r\n",
        "$$\r\n",
        "This explanation is mainly taken from [here](https://d2l.ai/chapter_multilayer-perceptrons/numerical-stability-and-init.html).\r\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vm5fWWd6j7Y1"
      },
      "source": [
        "## Initialization with transfer function\r\n",
        "Let's derive the optimal gain for LeakyReLU following a similar steps,\r\n",
        "\r\n",
        "$$\r\n",
        "f(x)=\\left\\{\\begin{array}{ll}\r\n",
        "a x & \\text { for } x<0 \\\\\r\n",
        "x & \\text { for } x \\geq 0\r\n",
        "\\end{array}\\right.\r\n",
        "$$\r\n",
        "\r\n",
        "Considering a single layer with activation gives, \r\n",
        "\r\n",
        "\r\n",
        "The expectation of the output is still zero but the variance changes and assuming the probability $P(x < 0) = 0.5$\r\n",
        "\r\n",
        "\\begin{split}\\begin{aligned}\r\n",
        "    \\mathrm{Var}[f(o_i)] = E[f(o_i)^2] & = \\frac{\\mathrm{Var}[o_i] + a^2 \\mathrm{Var}[o_i]}{2} = \\frac{1+a^2}{2}n_\\mathrm{in} \\sigma^2 \\gamma^2\r\n",
        "\\end{aligned}\\end{split}\r\n",
        "\r\n",
        "Therefore following the rest of derivation as before,\r\n",
        "\r\n",
        "$$\r\n",
        "\\sigma = gain\\sqrt{\\frac{2}{n_\\mathrm{in} + n_\\mathrm{out}}}, \\quad gain = \\sqrt{\\frac{2}{1+a^2}}\r\n",
        "$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8LxZCVS5pCsD"
      },
      "source": [
        "## Exercise 7: Best gain for Xavier Initialization with Leaky ReLU\r\n",
        "You're probably running out of time, so let me explain what's happening here. We derived a theoretical gain for initialization. But the question is whether it holds in practice? Here we have a setup to confirm our finding. We will try a range of gains and see the empirical optimum and whether it matches our theoretical value!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HqeL_3FuHn3X"
      },
      "source": [
        "N = 10 # number of trials\r\n",
        "gains = np.linspace(1/N, 3.0, N)\r\n",
        "test_accs = []\r\n",
        "train_accs = []\r\n",
        "for gain in gains:\r\n",
        "\r\n",
        "  def init_weights(m):\r\n",
        "    if type(m) == nn.Linear:\r\n",
        "        torch.nn.init.xavier_normal_(m.weight, gain)\r\n",
        "        # torch.nn.init.xavier_uniform_(m.weight, gain)\r\n",
        "\r\n",
        "  negative_slope = 0.1\r\n",
        "  actv = 'LeakyReLU(%f)'%negative_slope\r\n",
        "  net = Net(actv, 3*32*32, [128, 64, 32], 3).to(dev) \r\n",
        "  net.apply(init_weights)\r\n",
        "  criterion = nn.CrossEntropyLoss()\r\n",
        "\r\n",
        "  optimizer = optim.SGD(net.parameters(), lr=1e-2)\r\n",
        "  train_acc, test_acc = train_test_classification(net, criterion, optimizer,\r\n",
        "                                                  img_train_loader, img_test_loader,\r\n",
        "                                                  num_epochs=1, verbose=True)\r\n",
        "  test_accs += [test_acc]\r\n",
        "  train_accs += [train_acc]\r\n",
        "\r\n",
        "best_gain = gains[np.argmax(train_accs)]\r\n",
        "plt.plot(gains, test_accs, label='Test acc')\r\n",
        "plt.plot(gains, train_accs, label='Train acc')\r\n",
        "plt.scatter(best_gain, max(train_accs), label='best gain = %.1f'%best_gain, c='r')\r\n",
        "theoretical_gain = np.sqrt(2.0 / (1 + negative_slope ** 2))\r\n",
        "plt.scatter(theoretical_gain, max(train_accs), label='theoretical gain = %.2f'%theoretical_gain, c='g')\r\n",
        "plt.legend()\r\n",
        "plt.plot()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZMxtE7y0AF1o"
      },
      "source": [
        "---\n",
        "# Conclusion"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9x3YnWphApIa",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Wrapping Up Week 3\n",
        "\n",
        "try: t20;\n",
        "except NameError: t20=time.time()\n",
        "\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"Jzc9Ua0isYI\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uyHy54ZxPjPt"
      },
      "source": [
        "# Submit your responses\r\n",
        "Please run the following cell and then press \"Submit\" so we can record your responses."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "JAiufZ39Lpq8"
      },
      "source": [
        "import time\r\n",
        "import numpy as np\r\n",
        "from IPython.display import IFrame\r\n",
        "#@markdown #Run Cell to Show Airtable Form\r\n",
        "#@markdown ##**Confirm your answers and then click \"Submit\"**\r\n",
        "\r\n",
        "def prefill_form(src, fields: dict):\r\n",
        "  '''\r\n",
        "  src: the original src url to embed the form\r\n",
        "  fields: a dictionary of field:value pairs,\r\n",
        "  e.g. {\"pennkey\": my_pennkey, \"location\": my_location}\r\n",
        "  '''\r\n",
        "  prefills = \"&\".join([\"prefill_%s=%s\"%(key, fields[key]) for key in fields])\r\n",
        "  src = src + prefills\r\n",
        "  src = \"+\".join(src.split(\" \"))\r\n",
        "  return src\r\n",
        "\r\n",
        "\r\n",
        "#autofill time if it is not present\r\n",
        "try: t1;\r\n",
        "except NameError: t1 = time.time()\r\n",
        "try: t2;\r\n",
        "except NameError: t2 = time.time()\r\n",
        "try: t3;\r\n",
        "except NameError: t3 = time.time()\r\n",
        "try: t4;\r\n",
        "except NameError: t4 = time.time()\r\n",
        "try: t5;\r\n",
        "except NameError: t5 = time.time()\r\n",
        "try: t6;\r\n",
        "except NameError: t6 = time.time()\r\n",
        "try: t7;\r\n",
        "except NameError: t7 = time.time()\r\n",
        "\r\n",
        "#autofill fields if they are not present\r\n",
        "#a missing pennkey and pod will result in an Airtable warning\r\n",
        "#which is easily fixed user-side.\r\n",
        "try: my_pennkey;\r\n",
        "except NameError: my_pennkey = \"\"\r\n",
        "\r\n",
        "try: my_pod;\r\n",
        "except NameError: my_pod = \"Select\"\r\n",
        "\r\n",
        "try: w3_with_non_linear_loss;\r\n",
        "except NameError: w3_with_non_linear_loss = \"\"\r\n",
        "\r\n",
        "try: w3_poly_OoD;\r\n",
        "except NameError: w3_poly_OoD = \"\"\r\n",
        "\r\n",
        "try: w3_why_three;\r\n",
        "except NameError: w3_why_three = \"\"\r\n",
        "\r\n",
        "try: w3_fast_NTK;\r\n",
        "except NameError: w3_fast_NTK = \"\"\r\n",
        "\r\n",
        "try: w3_isHierarchical;\r\n",
        "except NameError: w3_isHierarchical = \"\"\r\n",
        "\r\n",
        "try: w3_whichActiv;\r\n",
        "except NameError: w3_whichActiv = \"\"\r\n",
        "\r\n",
        "times = np.array([t2,t3,t4,t5,t6,t7])-t1\r\n",
        "\r\n",
        "fields = {\"pennkey\": my_pennkey,\r\n",
        "          \"pod\": my_pod,\r\n",
        "          \"w3_with_non_linear_loss\":w3_with_non_linear_loss,\r\n",
        "          \"w3_poly_OoD\": w3_poly_OoD,\r\n",
        "          \"w3_why_three\":w3_why_three,\r\n",
        "          \"w3_fast_NTK\": w3_fast_NTK,\r\n",
        "          \"w3_isHierarchical\":w3_isHierarchical,\r\n",
        "          \"w3_whichActiv\":w3_whichActiv,\r\n",
        "          \"cumulative_times\": times}\r\n",
        "\r\n",
        "src = \"https://airtable.com/embed/shrElsLHM2gDYmFnl?\"\r\n",
        "\r\n",
        "#now instead of the original source url, we do: src = prefill_form(src, fields)\r\n",
        "display(IFrame(src = prefill_form(src, fields), width = 800, height = 400))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0kglwCDy7x71"
      },
      "source": [
        "## Feedback\r\n",
        "How could this session have been better? How happy are you in your group? How do you feel right now?\r\n",
        "\r\n",
        "Feel free to use the embeded form below or use this link:\r\n",
        "<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://airtable.com/shrNSJ5ECXhNhsYss\">https://airtable.com/shrNSJ5ECXhNhsYss</a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8_t2sxaejuMI"
      },
      "source": [
        "display(IFrame(src=\"https://airtable.com/embed/shrNSJ5ECXhNhsYss?backgroundColor=red\", width = 800, height = 400))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yGTLB5JFJ4-k"
      },
      "source": [
        "# Homeworks\r\n",
        "* (1) Join the Kaggle Competition to solve Animal Faces with random permutations using MLPs.\r\n",
        "* (2) Something where you debug something: Cross-entropy optimization with poor initialization, producing NaNs.\r\n",
        "* (3) Something related to ethics: A classification system with interest bias?\r\n",
        "* (4) Read some cool original paper:  Kernel vs Rich regimes paper?\r\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EPmj2vUfv4t7"
      },
      "source": [
        "# Kaggle competition\r\n",
        "https://www.kaggle.com/c/permuted-animal-faces/"
      ]
    }
  ]
}